Dance Box

ECE 5725 Final Project Report
By Mei-Jen Lee(ml2298), Ziyin Wang(zw252).

Demonstration Video

Project Objective

In this project, we designed a dance box that enables a user to fit into the box shown on the PiTFT screen. We used a Pi camera to capture body movements and lit a number of boxes, each lasting for several seconds, and we set different difficulty levels for users to play. A song is played in the background as the game starts and the score is displayed.

Introduction

Design and Testing

1. Install and test Pi camera

We installed a Raspberry Pi Camera through the CSI port and used the ‘raspistill’ command to take a picture. At first, we encountered an issue with the camera, which shows the message camera is not enabled in this build. Then, we solved this issue by using ‘sudo raspi-config’ to enable the camera. We also encountered an issue where the camera was not detected. We tried to reinstall the camera and reboot the system, after that the camera could be detected successfully. To figure out the suitable method of Pi camera, we tested three approaches, VideoCapture, WebcamVideoStream, and PiVideoStream. The results show that VideoCapture is the slowest, PiVideoStream has some loss of accuracy but it is the fastest. For a better user experience, we made a tradeoff and chose PiVideoStream.

2. Body object detection

We detected the position of the bounding box using haar feature-based cascade classifier (haarcascade_fullbody.xml) in the opencv. After loading the classifier, we called the detectMultiScale() function and adjusted the parameters to make the result more accurate. We mainly considered parameters of minNeighbor, scaleFactor and minSize. MinNeighbor defines the minimum number of neighbors that each target should be detected to be considered as a real target, we set it to 1 in this project. ScaleFactor represents the proportion of reduction of each image size, we made a tradeoff as it influences speed and accuracy. MinSize indicates the limitation of object size, we avoided detecting impossible body objects that are too short or too thin.

3. Play background music

We loaded a background music file, then called pygame.mixer.music.play() to play the music. The first argument loops determines how many times the music should be repeated, the second argument start indicates the time that the music begins to play, we set it to 0.0 so the music file should be played from the beginning. To stop playing the music, we used pygame.mixer.music.stop().

4. Screen pixel calculation

The coordinates are shown in Figure 1. The blue frame is the initial adjustment frame, the pink frame is the body object.

Generic placeholder image
Figure 1. Screen pixel calculation

5. Display camera frames

We displayed the camera frames on the PiTFT screen through pygame. The frames appeared inverted initially, so we used numpy.fliplr() and rot90() to reverse the order and rotate the array by 90 degrees. We used cv2.cvtColor() to convert the BGR color to RGB to fit pygame’s requirement. Then, we made a new surface and showed it on the screen.

6. Program FSM

This is the structure of the whole program, we defined 7 levels as shown in Figure 2. We set 4 physical buttons on PiTFT. 17, 22, 23, 27 represent for start, restart, score, quit respectively.

Generic placeholder image
Figure 2. Program FSM
Level 1 Show the game title and wait until game start
Level 2 Select difficulty
Level 3 Body calibration
Level 4 Present “Ready…” for 1 sec
Level 5 Present “Go!” for 1 sec
Level 6 Game (two difficulty levels)
Level 7 Show score
Table 1. Levels of Program FSM

7. Calibration FSM

This is the structure of the second layer FSM in level 3. Players can adjust their standing positions in this process.

Generic placeholder image
Figure 3. Calibration FSM
State 0 Draw red frame
Go to state 1 if detected in the frame
State 1 Draw orange frame
Wait for 0.9 sec and go to state 2 if detected in the frame more than 10 times in 0.9 sec
Else go back to state 0
State 2 Draw yellow frame
Wait for 0.9 sec and go to state 3 if detected in the frame more than 10 times in 0.9 sec
Else go back to state 0
State 3 Draw green frame
Wait for 0.9 sec and go to level 3 if detected in the frame more than 10 times in 0.9 sec
Else go back to state 0
Table 2. States of Calibration FSM

We generated initial random numbers for random frame assignment in state 3 because we want to avoid multiple generate before getting points. In each state, we have to record the start time when the state is about to change (like the transition line in the picture for it only happens once) instead of recording the start time in the beginning of the next state, because we don’t want to update start time again and again.

8. Game FSM

This is the structure of the second layer FSM in level 6.

Generic placeholder image
Figure 4. Game FSM
State 0 Draw red frame
Go to state 1 if detected in the frame
State 1 Draw yellow frame
Wait for 0.9 sec and go to state 2 if detected in the frame more than 10 times in 0.9 sec
Else go back to state 0
State 2 Display “Good!” in green
Wait for 0.9 sec and get 1 point if detected in the frame more than 10 times in 0.8 sec
Else go back to state 0
Table 3. States of Game FSM

9. Adjust difficulty

In the hard mode, we shrunk the box size to increase the difficulty during the game. Each time, the width decreases by 3 pixels and the height decreases by 2 pixels. We set a minimum width and height to avoid unlimited shrinking.

Generic placeholder image
Figure 5. Adjust box size

10. Hit definition

Generic placeholder image
Figure 6. Hit definition

We generated a coin and a mosquito in the game. The player would get bonus points if grabs the coin and would lose some points if the mosquito catches up with him. To define whether the mosquito hits the player, we used the center coordinates of the mosquito and calculated if the object is inside the box of the player. The mosquito keeps track of the player through the center coordinates, the length of vector between mosquito and player would cause high speed, so we used unit vector to update the coordinates of mosquito.

Result

In this project, we realized all the functionalities in the game as planned. We spent some time debugging the problems, and we solved all the problems successfully. We installed a Pi camera, and compared three methods to figure out the fastest one. We detected the body object and considered the impact of different parameters to get a more accurate result. We implemented the 7 layer FSM as the main structure of the game, and in the second layer, we have calibration FSM and game FSM. We provided two difficulty levels of the game, in the hard mode, the difficulty can be adjusted automatically.

Conclusions

In conclusion, we designed a dance box game. Users can fit into the box that is shown on the PiTFT screen. A background music is played in the game, and the remaining time of the game is dependent on the music and is shown on the screen. We used a Pi camera to capture body movements and displayed boxes on the screen, each box lasting for 5 seconds. When the timer runs out, the program will calculate the scores by counting the number of keypoints of the human body inside the box. We designed an algorithm to calculate the positions of the boxes and drew them on the video output of the PiTFT. The boxes appear on the screen randomly, and we added the requirement that the places be low and large enough for players to fit in. We created a user interface that allows users to select different difficulty levels and start the game. We added bonus points and obstacles that show on the screen for a limited time for players to collect and avoid. When a user receives higher/lower scores, the level is automatically adjusted up/down.

Future work

We plan to add some new rules for the game. For example, each player will have 3 lives, when a player gets a score lower than 20, he will lose one life and start a new round. When all 3 lives are lost, the game ends. We could also support multiple players, and record the highest score of each user.

Work Distribution

Mei-Jen Lee

ml2298@cornell.edu

Designed the overall software architecture.

Ziyin Wang

zw252@cornell.edu

Tested the overall system.

Parts List

All are provided in lab

References

Haarcascade_fullbody.xml
Cascade classifier
Display camera frames
Pygame play music
Start music
Level clear music
Game music
Game picture

Code Appendix

import random
  import RPi.GPIO as GPIO
  import numpy as np
  import time
  import cv2
  from imutils.video import WebcamVideoStream
  from imutils.video.pivideostream import PiVideoStream
  from pydub import AudioSegment
  from pydub.playback import play
  import threading
  
  import imutils
  import time
  import os
  import pygame
  
  body_classifier = cv2.CascadeClassifier('model/haarcascade_fullbody.xml')
  os.putenv('SDL_VIDEODRIVER','fbcon')
  os.putenv('SDL_FBDEV','/dev/fb1')
  os.putenv('SDL_MOUSEDRV','TSLIB')
  os.putenv('SDL_MOUSEDEV','/dev/input/touchscreen')
  TIMEOUT=240
  START=False
  QUIT=False
  b1=17
  b2=22
  b3=23
  b4=27
  GPIO.setmode(GPIO.BCM)
  GPIO.setup(b1,GPIO.IN, pull_up_down=GPIO.PUD_UP)
  GPIO.setup(b2,GPIO.IN, pull_up_down=GPIO.PUD_UP)
  GPIO.setup(b3,GPIO.IN, pull_up_down=GPIO.PUD_UP)
  GPIO.setup(b4,GPIO.IN, pull_up_down=GPIO.PUD_UP)
  
  CODERUN=True
  LEVEL="level1"
  REMAIN_TIME=31
  pygame.init()
  pygame.display.init()
  pygame.mouse.set_visible(True)
  
  #color definition
  black=(0,0,0)
  COLOR=black
  white=(255,255,255)
  GREEN=(0,255,0)
  ORANGE=(0,126,255)
  RED=(0,0,255)
  YELLOW=(0,255,255)
  
  #PiTFT screen setting
  (width, height)=(320,240)
  size=(width,height)
  screen = pygame.display.set_mode(size)
  
  #Image load
  ball = pygame.image.load("picture/minion.jpg")
  ballrect=ball.get_rect(center=(160,120))
  
  speed=[-1,-1]
  bug = pygame.image.load("picture/bug.png")
  bug = pygame.transform.scale(bug,(40,40))
  bugrect=bug.get_rect(center=(100,100))
  
  block = pygame.image.load("picture/block.jpeg")
  block = pygame.transform.scale(block,(50,50))
  blockrect=block.get_rect(center=(100,100))
  
  #font initialization
  EASY_POSITION=(120,100)
  HARD_POSITION=(120,170)
  myFont = pygame.font.SysFont("Times New Roman",30)
  myFont_small = pygame.font.SysFont("Times New Roman",22)
  difficulty_font = myFont.render("Select Difficulty:",20,white)
  easy_font = myFont.render("Easy",140,white)
  hard_font = myFont.render("Hard",140,white)
  easy = easy_font.get_rect(center=EASY_POSITION)
  hard = hard_font.get_rect(center=HARD_POSITION)
  
  frame=white
  COUNT_KEEP=0
  TEXT=""
  KEEPSTATE=0
  THRESHOLD=1
  CONFIGURATION_TIME=0.25
  GAMESTATE=0
  BEAT=39
  SCORE=0
  THRESHOLD2=1
  GAMELEVEL="level5"
  MODE_WIDTH=150
  MODE_HEIGHT=230
  MUSIC_LENGTH=32
  WIDTH_LIMIT=90
  HEIGHT_LIMIT=130
  EASY_X_MAX=320-MODE_WIDTH
  EASY_BOTTOM_MIN=229
  HARD_X_MAX=240-WIDTH_LIMIT-1
  HARD_BOTTOM_MIN=238
  STRIDE=45
  DIFFICULTY="Easy"
  
  
  try:
      pygame.mixer.music.load('music/mii.mp3')
      pygame.mixer.music.play(-1,0.0)
      print("Setting camera")
      #cap = WebcamVideoStream(src=-1).start()   
      cap = PiVideoStream().start()   
      time.sleep(1)
      STARTTIME=time.time()
      while CODERUN:
          #TIMEOUT
          NOW=time.time()
          if NOW-STARTTIME>TIMEOUT:
              print("time out")
              quit()
          #quit
          if not GPIO.input(b4):
              cap.stop()
              cv2.destroyAllWindows()
              quit()
          time.sleep(0.01)
          #WAIT UNTIL START
          if LEVEL=="level1":
              #select difficulty level    
              if not GPIO.input(b1):
                  pygame.mixer.music.load('music/mii.mp3')
                  pygame.mixer.music.play(-1,0.0)
                  LEVEL="level1.5"
              font1 = myFont.render("press button 17 to start",50,(0,0,0))
              font2 = myFont.render("Dance Box",120,(0,0,0))
              screen.fill(black)
              screen.blit(ball,ballrect)
              screen.blit(font1,(20,200))
              screen.blit(font2,(100,60))
              pygame.display.flip()
          #select difficulty
          elif LEVEL=="level1.5":
              SCORE=0
              for event in pygame.event.get():
                  #easy
                  if event.pos[0]<easy.right and event.pos[0]>easy.left and event.pos[1]>easy.top and event.pos[1]<easy.bottom:
                      print("select easy")
                      MODE_WIDTH=150
                      MODE_HEIGHT=230
                      X_MAX=EASY_X_MAX
                      BOTTOM_MIN=EASY_BOTTOM_MIN
                      GAMELEVEL="level5"
                      DIFFICULTY="Easy"
                      LEVEL="level2"
                  #hard
                  if event.pos[0]<hard.right and event.pos[0]>hard.left and event.pos[1]>hard.top and event.pos[1]<hard.bottom:
                      print("select hard")
                      MODE_WIDTH=150
                      MODE_HEIGHT=230
                      X_MAX=HARD_X_MAX
                      BOTTOM_MIN=HARD_BOTTOM_MIN
                      GAMELEVEL="level5.5"
                      DIFFICULTY="Hard"
                      LEVEL="level2"
              screen.fill(black)
              screen.blit(difficulty_font,(50,40))
              screen.blit(easy_font,EASY_POSITION)
              screen.blit(hard_font,HARD_POSITION)
              pygame.display.flip()
          #CAMERA CALIBRATION
          elif LEVEL=="level2":
              #select difficulty level    
              #read from camera
              frame = cap.read()
              #preprocessing for body detection
              frame = imutils.resize(frame,width=320)
              frame = cv2.flip(frame,1)
              gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)
              #body detection
              bodies = body_classifier.detectMultiScale(frame,scaleFactor=1.009,minNeighbors=1,minSize=(60,110))
              x,y,w,h=0,0,0,0    
              if len(bodies)>0:
                  (x,y,w,h) = bodies[0]
  
              COLOR=RED 
              #COUNT STAY TIME
  
  
              if KEEPSTATE==0:
                  COUNT_KEEP=0
                  TEXT=""
                  if (x>100 and y>1 and x+w<220 and y+h<239):
                      KEEPSTATE=1
                      KEEP_START=time.time()
              elif KEEPSTATE==1:
                  TIME_INTERVAL=time.time()-KEEP_START
                  if TIME_INTERVAL<CONFIGURATION_TIME:
                      TEXT="3"
                      if (x>100 and y>1 and x+w<220 and y+h<239):
                          COUNT_KEEP+=1
                      COLOR=ORANGE
                  else:
                      if COUNT_KEEP>=THRESHOLD:
                          KEEPSTATE=2
                          COUNT_KEEP=0
                          KEEP_START=time.time()
                      else:
                          KEEPSTATE=0
              elif KEEPSTATE==2:
                  TIME_INTERVAL=time.time()-KEEP_START
                  if TIME_INTERVAL<CONFIGURATION_TIME:
                      TEXT="2"
                      if (x>100 and y>1 and x+w<220 and y+h<239):
                          COUNT_KEEP+=1
                      COLOR=YELLOW
                  else:
                      if COUNT_KEEP>=THRESHOLD:
                          KEEPSTATE=3
                          COUNT_KEEP=0
                          KEEP_START=time.time()
                      else:
                          KEEPSTATE=0
              elif KEEPSTATE==3:
                  TIME_INTERVAL=time.time()-KEEP_START
                  if TIME_INTERVAL<CONFIGURATION_TIME:
                      TEXT="1"
                      if (x>100 and y>1 and x+w<220 and y+h<239):
                          COUNT_KEEP+=1
                      COLOR=GREEN
                  else:
                      if COUNT_KEEP>=THRESHOLD:
                          COUNT_KEEP=0
                          KEEPSTATE=0
                          #start to count temple
                          TEMPLE=0
                          LEVEL="level3"
                          LEVEL3START=time.time()
                          target_x = random.randrange(1, X_MAX, STRIDE)
                          target_bottom = random.randrange(BOTTOM_MIN, 239, STRIDE)
                          target_w = MODE_WIDTH
                          target_h = MODE_HEIGHT
                          pygame.mixer.music.stop()
                          pygame.mixer.music.load('music/mario.mp3')
                          pygame.mixer.music.play(0,0.0)
                      else:
                          KEEPSTATE=0
                 
              cv2.putText(frame,TEXT,(120,160),cv2.FONT_HERSHEY_DUPLEX, 5, COLOR,9)
              cv2.putText(frame,"Keep your body in the frame",(8,20),cv2.FONT_HERSHEY_DUPLEX, 0.6,COLOR,1)
              cv2.rectangle(frame,(100,1),(100+120,1+238),COLOR,2)
              frame = np.fliplr(frame)
              frame = np.rot90(frame)
              frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
              surf = pygame.surfarray.make_surface(frame)
              screen.fill(black)
              screen.blit(surf,(0,0))
              pygame.display.flip()
              #cv2.imshow('haha',frame)
              if cv2.waitKey(1)==13:
                  break
          #GAME START!
          #font size
          elif LEVEL=="level3":
              if time.time()-LEVEL3START>1:
                  LEVEL="level4"
                  LEVEL4START=time.time()
              font1 = myFont.render("Ready...",20,black)
              font2 = myFont.render(DIFFICULTY+" Level",20,black)
              screen.fill(black)
              screen.blit(ball,ballrect)
              screen.blit(font1,(120,160))
              screen.blit(font2,(80,40))
              pygame.display.flip()
          elif LEVEL=="level4":
              if time.time()-LEVEL4START>1:
                  LEVEL=GAMELEVEL
                  MUSIC_START=time.time()
              font1 = myFont.render("Go",20,black)
              font2 = myFont.render(DIFFICULTY+" Level",20,black)
              screen.fill(black)
              screen.blit(ball,ballrect)
              screen.blit(font1,(130,160))
              screen.blit(font2,(80,40))
              pygame.display.flip()
  
          elif LEVEL=="level5":
              #select difficulty level    
              if not GPIO.input(b1):
                  pygame.mixer.music.load('music/mii.mp3')
                  pygame.mixer.music.play(-1,0.0)
                  LEVEL="level1"
              #restart
              if not GPIO.input(b2):
                  pygame.mixer.music.load('music/mario.mp3')
                  pygame.mixer.music.play(0,0.0)
                  LEVEL="level3"
  
              #show score
              if not GPIO.input(b3):
                  pygame.mixer.music.load('music/gameover.mp3')
                  pygame.mixer.music.play(0,0.0)
                  LEVEL="level6"
              
              game_time = time.time()-MUSIC_START
              REMAIN_TIME=int(31-game_time)
              if time.time()-MUSIC_START>31:
                  LEVEL="level6"
                  pygame.mixer.music.load('music/gameover.mp3')
                  pygame.mixer.music.play(0,0.0)
              #read from camera
              frame = cap.read()
              #preprocessing for body detection
              frame = imutils.resize(frame,width=320)
              frame = cv2.flip(frame,1)
              gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)
              #body detection
              bodies = body_classifier.detectMultiScale(frame,scaleFactor=1.009,minNeighbors=1,minSize=(60,110))
              x,y,w,h=0,0,0,0    
              if len(bodies)>0:
                  (x,y,w,h) = bodies[0]
              #for (x,y,w,h) in bodies:
              #    cv2.rectangle(frame,(x,y),(x+w,y+h),COLOR,2)
              
              #block fly
              blockrect=blockrect.move(speed)
              if blockrect.left<0 or blockrect.right>width:
                  speed[0]=-speed[0]
              
              if blockrect.top<0 or blockrect.bottom>height:
                  speed[1]=-speed[1]
              BLOCK_POSITION_X = blockrect.center[0]
              BLOCK_POSITION_Y = blockrect.center[1]
              #Hit Bug
              if len(bodies)>0:
                  if BLOCK_POSITION_X>x and BLOCK_POSITION_Y>y and BLOCK_POSITION_X<x+w and BLOCK_POSITION_Y<y+h:
                      SCORE+=0.2
              
              #text position
              POSITION=(120,160)
              #font size
              SIZE=5
              #red
              #COUNT STAY TIME
              if KEEPSTATE==0:
                  TEXT="0"
                  COLOR=RED
                  COUNT_KEEP=0
                  TEXT=""
                  if (x>target_x and y>target_bottom-target_h and x+w<target_x+target_w and y+h<target_bottom):
                      KEEPSTATE=1
                      KEEP_START=time.time()
              #WAIT FOR ONE SECOND
              elif KEEPSTATE==1:
                  COLOR=YELLOW
                  TEXT="Keep"
                  TIME_INTERVAL=time.time()-KEEP_START
                  if TIME_INTERVAL<CONFIGURATION_TIME:
                      if (x>target_x and y>target_bottom-target_h and x+w<target_x+target_w and y+h<target_bottom):
                          COUNT_KEEP+=1
                  else:
                      if COUNT_KEEP>=THRESHOLD2:
                          KEEPSTATE=2
                          COUNT_KEEP=0
                          KEEP_START=time.time()
                      else:
                          KEEPSTATE=0
              #GET POINT!
              elif KEEPSTATE==2:
                  #Green
                  COLOR=GREEN
                  TEXT="Good!"
                  TIME_INTERVAL=time.time()-KEEP_START
                  if TIME_INTERVAL>0.5:
                      KEEPSTATE=0
                      SCORE+=10
                      target_x = random.randrange(1, X_MAX, STRIDE)
                      target_bottom = random.randrange(BOTTOM_MIN, 239, STRIDE)
                      target_w = MODE_WIDTH
                      target_h = MODE_HEIGHT
                 
              #Score
              cv2.putText(frame,"Score:"+str(int(SCORE)),(190,30),cv2.FONT_HERSHEY_DUPLEX, 0.8,(0,0,0),2)
              #Remain time
              cv2.putText(frame,"Time:"+str(REMAIN_TIME),(10,30),cv2.FONT_HERSHEY_DUPLEX, 0.8,(0,0,0),2)
              cv2.putText(frame,TEXT,(60,150),cv2.FONT_HERSHEY_DUPLEX, 2, COLOR,8)
              cv2.rectangle(frame,(target_x,target_bottom-target_h),(target_x+target_w,target_bottom),COLOR,2)
              
              frame = np.fliplr(frame)
              frame = np.rot90(frame)
              frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
              surf = pygame.surfarray.make_surface(frame)
              screen.fill(black)
              screen.blit(surf,(0,0))
              screen.blit(block,blockrect)
              pygame.display.flip()
              if cv2.waitKey(1)==13:
                  break
          #hard
          elif LEVEL=="level5.5":
              #select difficulty level    
              if not GPIO.input(b1):
                  pygame.mixer.music.load('music/mii.mp3')
                  pygame.mixer.music.play(-1,0.0)
                  LEVEL="level1"
              #restart
              if not GPIO.input(b2):
                  pygame.mixer.music.load('music/mario.mp3')
                  pygame.mixer.music.play(0,0.0)
                  LEVEL="level3"
              #show score
              if not GPIO.input(b3):
                  pygame.mixer.music.load('music/gameover.mp3')
                  pygame.mixer.music.play(0,0.0)
                  LEVEL="level6"
              
              time.sleep(0.02)
              game_time = time.time()-MUSIC_START
              REMAIN_TIME=31-game_time
              if time.time()-MUSIC_START>31:
                  LEVEL="level6"
                  pygame.mixer.music.load('music/gameover.mp3')
                  pygame.mixer.music.play(0,0.0)
              #body detection
              #read from camera
              frame = cap.read()
              #preprocessing for body detection
              frame = imutils.resize(frame,width=320)
              #have to flip for some reason
              frame = cv2.flip(frame,1)
              gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)
              #body detection
              bodies = body_classifier.detectMultiScale(frame,scaleFactor=1.009,minNeighbors=1,minSize=(60,110))
              x,y,w,h=0,0,0,0    
              if len(bodies)>0:
                  (x,y,w,h) = bodies[0]
              #for (x,y,w,h) in bodies:
                  #cv2.rectangle(frame,(x,y),(x+w,y+h),COLOR,2)
              
              BODY_CENTER = [x+w/2,y+h/2]
              #bug fly
              bugrect=bugrect.move(speed)
              
              BUG_POSITION_X = bugrect.center[0]
              BUG_POSITION_Y = bugrect.center[1]
              
              if BODY_CENTER[0]<BUG_POSITION_X: 
                  x_direction=-2
              else:
                  x_direction=2
  
              if BODY_CENTER[1]<BUG_POSITION_Y: 
                  y_direction=-2 
              else:
                  y_direction=2
              
              if bugrect.left<0 or bugrect.right>width:
                  speed[0]=-speed[0]
              
              if bugrect.top<0 or bugrect.bottom>height:
                  speed[1]=-speed[1]
               
              #if no collision && user detected -> than chase users
              if len(bodies)>0 and bugrect.left>0 and bugrect.right<width and bugrect.top>0 and bugrect.bottom<height:
                  speed=[x_direction,y_direction] 
  
  
              #Hit Bug
              if len(bodies)>0:
                  if BUG_POSITION_X>x and BUG_POSITION_Y>y and BUG_POSITION_X<x+w and BUG_POSITION_Y<y+h:
                      if SCORE>0.5:
                          SCORE-=0.5
              
              
              #text position
              POSITION=(120,160)
              #font size
              SIZE=5
              #COUNT STAY TIME
              if KEEPSTATE==0:
                  TEXT="0"
                  COLOR=RED
                  COUNT_KEEP=0
                  TEXT=""
                  if (x>target_x and y>target_bottom-target_h and x+w<target_x+target_w and y+h<target_bottom):
                      KEEPSTATE=1
                      KEEP_START=time.time()
              #WAIT FOR ONE SECOND
              elif KEEPSTATE==1:
                  COLOR=YELLOW
                  TEXT="Keep"
                  TIME_INTERVAL=time.time()-KEEP_START
                  if TIME_INTERVAL<CONFIGURATION_TIME:
                      if (x>target_x and y>target_bottom-target_h and x+w<target_x+target_w and y+h<target_bottom):
                          COUNT_KEEP+=1
                  else:
                      if COUNT_KEEP>=THRESHOLD2:
                          KEEPSTATE=2
                          COUNT_KEEP=0
                          KEEP_START=time.time()
                      else:
                          KEEPSTATE=0
              #GET POINT!
              elif KEEPSTATE==2:
                  #Green
                  COLOR=GREEN
                  TEXT="Good!"
                  POSITION=(70,150)
                  SIZE=3
                  TIME_INTERVAL=time.time()-KEEP_START
                  if TIME_INTERVAL>0.5:
                      KEEPSTATE=0
                      SCORE+=10
                      target_x = random.randrange(1, X_MAX, STRIDE)
                      target_bottom = random.randrange(BOTTOM_MIN, 239, STRIDE)
                      target_w = MODE_WIDTH
                      target_h = MODE_HEIGHT
                      if MODE_WIDTH>WIDTH_LIMIT+6:
                          MODE_WIDTH-=3
                      if MODE_HEIGHT>HEIGHT_LIMIT+2:
                          MODE_HEIGHT-=2
                 
              cv2.putText(frame,"Score:"+str(int(SCORE)),(190,30),cv2.FONT_HERSHEY_DUPLEX, 0.8,(0,0,0),2)
              #Remain time
              cv2.putText(frame,"Time:"+str(int(REMAIN_TIME)),(10,30),cv2.FONT_HERSHEY_DUPLEX, 0.8,(0,0,0),2)
              cv2.putText(frame,TEXT,(60,150),cv2.FONT_HERSHEY_DUPLEX, 2, COLOR,8)
              cv2.rectangle(frame,(target_x,target_bottom-target_h),(target_x+target_w,target_bottom),COLOR,2)
              
              frame = np.fliplr(frame)
              frame = np.rot90(frame)
              frame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
              surf = pygame.surfarray.make_surface(frame)
              screen.fill(black)
              screen.blit(surf,(0,0))
              screen.blit(bug,bugrect)
              pygame.display.flip()
              if cv2.waitKey(1)==13:
                  break
  
          #Show score
          elif LEVEL=="level6":
              
              
              for event in pygame.event.get():
                  if event.type==pygame.MOUSEBUTTONDOWN and event.pos[0]<320 and event.pos[0]>0 and event.pos[1]>0 and event.pos[1]<240:
                      pygame.mixer.music.load('music/mii.mp3')
                      pygame.mixer.music.play(-1,0.0)
                      LEVEL="level1"
  
              
                  #select difficulty level    
              #if not GPIO.input(b1):
              #    pygame.mixer.music.load('music/mii.mp3')
              #    pygame.mixer.music.play(-1,0.0)
              #    LEVEL="level1"
              #restart
              if not GPIO.input(b2):
                  pygame.mixer.music.load('music/mario.mp3')
                  pygame.mixer.music.play(0,0.0)
                  LEVEL="level3"
              font1 = myFont_small.render("press the screen to play again",30,(0,0,0))
              font2 = myFont_small.render("press button 22 to quit",30,(0,0,0))
              font3 = myFont.render("Score:"+str(int(SCORE)),100,(0,0,0))
              font4 = myFont.render(DIFFICULTY+" Level",50,black)
              screen.fill(black)
              screen.blit(ball,ballrect)
              screen.blit(font1,(20,170))
              screen.blit(font2,(20,200))
              screen.blit(font3,(100,60))
              screen.blit(font4,(100,30))
              pygame.display.flip()
  
              
  except KeyboardInterrupt:
      cap.stop()
      cv2.destroyAllWindows()
      quit()
              
  except:
      cap.stop()
      cv2.destroyAllWindows()
      quit()
              
  
  finally:
      cap.stop()
      cv2.destroyAllWindows()